Robotic manipulator with visual guidance &amp; tactile sensing

ABSTRACT

A robotic manipulator includes one or multiple end effectors that can engage with an object, and one or multiple cameras that simultaneously observe each end effector, and the surrounding environment. For example, an end effector can include a contact surface including tactile markers which can deform when the end effector contacts the object.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.63/240,285 filed Sep. 2, 2021, the entire contents of which are herebyincorporated for all purposes in their entirety.

BACKGROUND

Robotic grippers play an important role in the modern manufacturingindustry. Gripper and finger designs, sensing capability, andcorresponding technological developments have recently been the focus ofmany researchers and commercial companies over the globe to meet demandsof factory automation towards Industry 4.0.

Slippage detection, contact force estimation, and grasp control arefeatures of robotic grippers, and tactile sensing allows roboticgrippers to achieve robust grasping and successful object manipulation.Several types of tactile sensors and methods have been addressed andintegrated with robotic fingers to avail such features. Neuromorphicsensors have been used due to the increased expectation of robots onhigh precision requirements of tasks, timely detection of transientchanges in dynamic scenes, and efficient acquisition and processing ofsensory information enabling real-time response.

Neuromorphic vision-based tactile sensing holds promises to highprecision robotic manipulation task requirements in industrialmanufacturing and household services. In existing systems, visionsensors can be placed within the gripper's fingers or jaws to clearlycapture the tactile or visual activity at the fingertips. However, sucha camera placement can cause problems related to the safety andperformance of the system. For example, the camera wiring and structurecan restrict the movement of the gripper, the gripper operation canaffect the camera performance (e.g., due to vibration from the gripperthat is translated to the camera), or the camera can be damaged by thegripper's movement (e.g., the camera can be damaged if the grippercollides with an object).

A finger design that suits grippers for operations while achievingeffective sensing is needed. Apart from that, the fingertip as aninterface can play a helpful role in tactile sensing as well as inhandling a wide variety of targets/objects. Thus, a novel robotic fingerthat serves multiple purposes, enhances tactile sensing capability, andoffers a modular approach to replace fingertips to handle a widecategory of objects/targets is an attractive option in robotic grippersand very much needed in industrial applications.

Soft robotic grippers have gained traction over the last few years owingto recent breakthroughs in multiple science and engineering disciplines.Unlike conventional grippers that are composed of rigid links andjoints, soft grippers utilize flexible and compliant materials, makingthem a better candidate in unstructured environments and in handlingdelicate and fragile objects as compared to rigid grippers. Thecompliance and conformity of soft grippers allow them to envelop objectsof different sizes and shapes while holding them, offering considerableadvantage over conventional grippers in various applications.

Soft robots can obtain constantly updated information about theirinternal state (proprioception) and external tactile sensing(exteroception) to achieve robust grasping and fine manipulation.However, the increased degrees of freedom (DOF) and the limited range ofsensors that can be used with them present significant challenges thathinder their perception abilities and limit their applications.Vision-based sensing is an active research area that investigateswhether a camera-based sensor can be utilized to acquire informationabout the gripper and its surroundings. Such sensors present a potentialsolution that can provide adequate proprioceptive and exteroceptiveinformation for soft robotic grippers and improve their grasping andmanipulation abilities.

Recently, event-based camera technology has emerged with its potentialto revolutionize robotic vision. Unlike frame-based traditional cameras,event-based cameras detect transient changes in dynamic scenes in termsof brightness intensity. Moreover, event-based cameras have a highertemporal resolution, lower latency, efficient data processingcapability, and consume less power as compared to frame-based cameras.While performing manipulation tasks, timely detection of proprioceptiveand exteroceptive features is helpful for robotic grippers/hands toeffectively regulate the grip force and maintain a stable grasp.Therefore, a novel robotic gripping system that incorporates a softcompliant finger and a neuromorphic event-based camera sensor to refinegrasping capabilities and observe proprioceptive and exteroceptiveinformation such that the robot is superiorly able to handle differenttypes of objects is an attractive option that is needed in variousapplications.

Recent developments in robotic technologies have made them a competitivechoice in a variety of industrial processes. Among other applications,precise robotic machining has been studied extensively by academics andpractitioners since robots offer significant advantages over CNCmachines in terms of flexibility, mobility, cost efficiency, andworkspace volumes. However, the relatively low stiffness of roboticmanipulators and the unstructured environment degrades the reliabilityand repeatability of robotic operation under contact forces and torques;and hence is a limiting factor in precise machining processes. As such,high-precision robotic machining remains an open area for research anddevelopment.

A well-known approach to resolve repeatability and precision challengesin robotic operation is to provide a closed-loop control mechanism thatactuates the system based on sensory feedback and perception algorithms.For machining purposes, these perception systems must convey accurateestimates on the position and orientation of the robot's machining toolalong with contact forces and torques. Existing technologies for roboticmachining separate the perception process into two sensory sub-systems;the first focuses on initial positioning of the machining tool viametrology (e.g., with laser trackers, cameras, etc.) while the secondmonitors contact forces and torques using a formation of tactile sensors(e.g., strain gauge, piezoelectric, etc.). While such configurations canprovide ample performance, the requirement of two sensory sub-systems onthe same machining tool significantly increases development cost andraises several issues of installation complexity, maintenance, powerconsumption, sensor synchronization, and data communication.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 is a schematic view of an example robotic manipulator includingmultiple tactile interfaces, in accordance with various embodiments;

FIG. 2A shows the robotic manipulator of FIG. 1 in accordance withvarious embodiments;

FIG. 2B shows the robotic manipulator of FIG. 2 in accordance withvarious embodiments;

FIG. 3 shows example tactile interfaces with sensed tactileillustrations for use with the robotic manipulator of FIG. 1 , inaccordance with various embodiments;

FIG. 4 shows an additional example robotic manipulator, in accordancewith various embodiments;

FIG. 5 shows a schematic view of the robotic manipulator of FIG. 4 , inaccordance with various embodiments;

FIG. 6A shows an example end effector for use with a roboticmanipulator, in accordance with various embodiments;

FIG. 6B shows example engagement surfaces for use with the end effectorof FIG. 6A, in accordance with various embodiments;

FIG. 7A shows the end effector of FIG. 6A prior to contacting an object,in accordance with various embodiments;

FIG. 7B shows the end effector of FIG. 6A after contacting the object,in accordance with various embodiments;

FIG. 8 shows an example robotic manipulator including multiple endeffectors of FIG. 6A, in accordance with various embodiments;

FIG. 9 shows another example robotic manipulator including a single endeffector of FIG. 6A, in accordance with various embodiments;

FIG. 10 shows an example end effector including an optical sensor and anengagement surface, in accordance with various embodiments;

FIG. 11 shows an example end effector with multiple engagement surfaces,in accordance with various embodiments;

FIG. 12 shows another example end effector with tactile sensors, inaccordance with various embodiments;

FIG. 13 shows an example end effector with tactile sensors and opticalsensors, in accordance with various embodiments;

FIG. 14A shows the end effector of FIG. 10 positioned near a workpiece,in accordance with various embodiments;

FIG. 14B shows the end effector of FIG. 10 aligned with the workpiece,in accordance with various embodiments; and

FIG. 14C shows the end effector of FIG. 10 positioned for machining ofthe workpiece, in accordance with various embodiments.

FIG. 15 shows a neural network that can be used for estimating dataassociated with the robotic manipulator.

DETAILED DESCRIPTION

Embodiments and techniques described herein are directed to roboticmanipulators with end effectors. The end-effector can be or include agripper, multi-figured hand, or a tool that includes actuators, sensorsand/or capabilities such as vision-based tactile sensing and visualguidance (e.g., using frame-based and event-based vision technologies).

There is a growing demand for enhanced sensing and control capabilitiesin robotic gripper systems to handle a variety of targets and tackle theuncertainties occurring during physical tasks. Neuromorphic visionsensors detect transient changes in dynamic scenes asynchronously at apixel-level with a high temporal resolution, low latency, and widedynamic range. Such event-based sensors are a potential alternative toconventional tactile sensors. Neuromorphic sensing and computingtechnologies can be applied to automotive, mobile, medical, industrial,and consumer sectors.

Sense of touch and vision are highly important sensory modalities thatcan enable a controlled grip. Neuromorphic sensors offer a viablesolution to emulate the processing of sensor signals from suchbiological modalities. The human hand is the most specialized part ofthe body that provides accurate tactile feedback. Detection of incipientslip is one function of the tactile sensing modalities which enablesrobust grasping and dexterous manipulation. During the tactile activity,signal patterns from different receptors are diverse for different tasksand their combination increases the level of pattern complexity.Difficulties in obtaining a clear model for such a complex biologicalsystem is one of the primary reasons for the limited progress inartificial tactile sensing and development of the neuromorphic tactilesensor. Alternatively, the neuromorphic approach can be used totransform tactile signals to biological relevant representation (spikeevents). Recently, drawing inspiration from the behavior ofmechanoreceptors in the human hand, some studies have demonstrated thefeasibility of a tactile-event driven model for grasp control anddeveloped a slip detection and suppression strategy for robotic hands.

Vision is one of the most important sensing modalities heavily used byhumans for perception. In fact, the retina is the extensively studiedhuman neuro-biological system which remains a prominent example for themodel, design, and implementation of neuromorphic sensors. Conventionalframe-based image sensors are focused on implementing the ‘what’ systemby which they neglect the dynamic information in the visual scene.Recently, Dynamic vision sensor (DVS) was mainly developed to realizethe ‘where’ system. The DVS sensor constitutes a simplified three-layermodel of the human retina that operates in continuous time by respondingto brightness changes in the scene. Each individual pixel in the sensorarray works autonomously and responds to temporal contrast by generatingasynchronous spiked events. Various examples of the present disclosureutilize such a sensor for tactile sensing to enhance sensitivity andhandle a wide class of targets/objects.

Tactile sensing can be used in robotic grasping and manipulation.Robotic grippers and hands can be equipped with different types oftactile sensors. Based on the working principles, tactile sensing can beachieved by detecting object motions directly or indirectly. The idea ofusing frame-based image sensors for tactile sensing is not new andtypically allows detecting of object motion. Detecting the internalreflections via optic-based and marker-based vision sensing, wheremarkers are placed on the skin surface and their displacements aremeasured using image processing techniques and registration of objectsthrough marker-less localization is achieved. A frame-based camera cantrack the markers on the inner surface of a soft fingertip in order toprovide information about the contact force distribution whiledeformation occurred. Often, vision sensors are placed underneath theskin surface to detect the motion of markers which somehow limits theability of the vision sensor in distinguishing whether the change ofcontacts is from a grasped object or external disturbances.

Machine learning methods can be used to estimate contact force andclassify materials for a grasp. In some examples, a sensor andcorresponding encoding methods can be used for texture classificationtasks. In some examples, implementations include an event-based camerawith eye-in-hand configuration to detect, extract, and track high-levelfeatures of a target object.

Soft grippers can boast high compliance, adaptability, and softness inphysical interactions that can enable soft grippers to handle a varietyof fragile and delicate objects and fulfill the application needs inhuman-centered and industrial environments. However, utilizing suchqualities and unlocking their potential face several difficultiesincluding challenging perception and sensing.

Humans show impressive capabilities of grasping unknown objects andperforming dexterous tasks in unstructured environments. When handlingan object, we acquire information about it using receptors and senseswhich enable us to apply the right amount of force to keep the objectintact even when we do not know its physical properties. Researchershave been striving to develop robotic grippers that mimic human graspingby improving the proprioceptive and exteroceptive tactile sensing andperception capabilities.

An event-based camera with a pixel resolution of 128×128 and a temporalresolution of 500 μs can be used to obtain tactile information from afingertip that contains marker dots. The camera can be capable ofdetecting the fast contact of an object with the fingertip, the contactposition, the object orientation, and the fast changes in slip byanalyzing the intensity changes in the marker dots. Neuromorphicvision-based sensors can be used to extract tactile information fromsoft skins and fingertips.

Robotic platforms can be associated with a high cost-efficiency andflexibility. A pertinent element in attaining this precision isobtaining high-quality sensor-based information during the machiningprocess to be utilized in intelligent decision-making and controlsystems.

Sensory systems for robotic machining can benefit from conveyinginformation on the position and orientation of the machining tool aswell as tactile feedback on contact forces and torques. The former typeof data can be inferred from metrology systems such as cameras and lasertrackers. Tactile information, such as information related to contactforces, can be useful for the success of precise and sensitive machiningoperations to guarantee repeatability and avoid damaging delicateworkpieces. Force Feedback controllers can be used for precise roboticmanipulation and machining and can yield increases in performance.Contact force data also plays an important role in evaluating thesuccess of drilling operations along with identifying hazardous orabnormal conditions. Tactile information (e.g., contact forces andtorques) are often inferred using an array of strain gauge orpiezoelectric sensors installed on the robot's end effector. In oneexample, an ATI's D330-30 strain gauge-based sensor installed betweenthe manipulator's end effector and a spindle was utilized in a feedbackcontroller to improve the quality of drilling processes and reducesliding movements. A JR3 160/50M force sensor can be used to estimateand control forces in 3D for more precise robotic drilling. Kirstler9257A piezoelectric sensors can be for monitoring forces and torquesduring the drilling process for performance evaluation. A novel sensorwith two groups of dynamometers can provide accurate estimation of axialforces on a drilling tool. All the aforementioned sensors provide ampleaccuracy and utilizing them in feedback algorithms has provenadvantageous to the overall quality of machining. However, most of thesesensors suffer from low-bandwidth and high latency. Additionally, in theabsence of direct contact, these sensors do not provide any informationon the external surroundings of the robot's end effector; as such, mostpractical configurations couple these sensors with other metrologysystems for initial positioning and/or obstacle detection. For example,such configurations may use two different sensors (e.g., the first ofwhich may be a force sensing load cell housed in the pressure foot toestimate contact forces and orientation, while the second sensor may bean external camera utilized for global positioning of the machiningtool). Such use of two sensory systems may boost development cost andcause several complexities with installation, maintenance, powerconsumption, sensor synchronization, and/or data communication.

Various aspects of the present disclosure build upon recent developmentsin optical tactile sensing to introduce a purely vision-based sensor forrobotic machining. Vision-based tactile sensing has demonstratedadvantages in cost, latency, and resolution over other alternatives. Inparticular, neuromorphic cameras offer microsecond level latency, a highdynamic range up to 120 db, and a very low power consumption, makingthem suitable for precise machining applications. Various aspects of thepresent disclosure also make use of the versatility of optical sensorsto introduce a full solution for robotic machining where a singleoptical sensor can observe both the external surroundings in additionalto the tactile interfaces/engagement surfaces. One example is presentedin FIG. 10 , for instance. The wide range of capabilities of the sensormakes it sufficient for the full operation of a machining robot, whichin turn can reduce cost and alleviate power consumption and complexityconcerns when compared to alternative solutions that require multiplesensory systems. Furthermore, the flexibility of visual sensors enablesmultiple design variants in terms of the numbers of contact points,camera placement, and sensor dimensions, such as showcased in FIGS. 11,12 and 13 . Finally, the sensor offers low-cost customization potential,as most components can be 3D-printed or casted to fit specificrequirements. For instance, several low-cost tactile interfaces can befabricated, which in turn offers an inexpensive trade-off betweenresolution and range for the contact force estimation problem.Similarly, the housing of the sensor can be 3D-printed at differentdimensions according to application requirements. As major manufacturersare building smaller cameras, the systems can be fabricated at verysmall sizes which widen possible applications to areas such assmall-scale electronics manufacturing and assembly, which arechallenging applications for current force/torque sensors due to theirrelatively bulky size.

The current event-based sensors are not available in miniature size.Moreover, the placements of the event camera at the finger level reducesthe workspace in manipulation operation and increases the possibility ofhitting objects in the workspace. Thus, a mirror optic system can engageneuromorphic tactile sensing with different fingertips suitable for aclass of objects, customizing sensitivity and range.

This disclosure generally relates to robotic grippers and tactilesensing, particularly, to a robotic finger with a sense of touch thatprovides sensory feedback capability for robotic grippers to robustlygrasp under uncertainties and handle a wide variety of objects with highprecision for automation.

In some examples of the present disclosure, a novel robotic finger canmeasure finger-tip tactile information with enhanced sensitivity andrange, with an integrated event-based camera. In various embodiments,one or more optic lenses can be placed at any suitable point of theoptical channel of the event-based camera. The position of the one ormore optic lenses can be based, for example, on the gripper-operationsand/or to fulfill the field of view requirements. In furtherembodiments, the robotic finger can include an illumination system thatcan provide customized lighting conditions for the optical sensing.

In the robotic finger systems described herein, fingers (e.g., fingersincluding finger-tip tactile material) can be used in combination withsoft, hard and multiple fingers to suit different applications, tohandle a variety of targets and to attain robust grasping and precisemanipulation. In various embodiments, materials for use with the fingercan be chosen based on, for example, the operation requirements and/orthe categories of objects to be handled. The use of the novel roboticfinger in systems allows for the system to be designed for cameraintegration, allows for a novel approach for modular finger-tip tactileinterfaces, and allows for the use of methods for measuring event-basedtactile information from different interfaces.

In embodiments described herein, (e.g., embodiments discussed inreference to FIGS. 1 and 6A) a novel way of sensorization of the fingerconsidering real-time operations, grasping applications, and safetyaspects in robotic grippers/hands is discussed.

In known robotic fingers, a vision sensor is placed simply at thebackside of the gripper-fingertip or used with the standalone tactilesensing module to capture the tactile activity. Our customizable designstructure (e.g., as shown in FIG. 1 ) engages optical channel/systemwithin the finger embodiment, facilitates the use of multiple tactileinterfaces at the fingertip, and overcomes the problems related to thesafety and performance of the vision-based tactile sensing. Inparticular, the camera placement poses the following safety-,workspace-, and performance-related concerns: i) Camera wiring andstructure that could possibly restrict the movement of thegripper-fingers; ii) vibrations transmitted from the fingers to thecamera while performing gripper operation that could degrade thevision-based tactile sensing performance; and iii) the finger couldpossibly collide with an object during robot operations and cause damageto the camera. Overall, the finger design provides an optical path tothe stationary camera to acquire the tactile activity during objecthandling. Such arrangements ensure camera-safety duringgripper-operation in unstructured environments and provide moreoperational space to the gripper.

The customizable finger-structure design (e.g., as shown in FIG. 6A)emphasizes the sensorization of a soft finger without negativelyaffecting the finger compliance and conformity. Additionally, thedeveloped finger-structure facilitates the use of multiple tactileinterfaces (e.g., as shown in FIG. 6B) to interact with objects/targets.Known gripping systems don't have or utilize a passive soft finger withneuromorphic vision-based sensing, different tactile interfaces, andskin- and joint-markers to superiorly extract proprioceptive andexteroceptive information. Employing such a finger in robotic grippers,for example the robotic grippers described herein, allows pinch andenvelope grasp behaviors and at the same time ensure safe, adaptive, andprecise handling of a wide variety of objects in both human-centered andunstructured environments.

FIG. 1 is a schematic view of an example robotic manipulator 100 thatcan include multiple tactile interfaces 104. The robotic manipulator 100can include an event camera 106, an optic mirror system 102, and atactile interface 104. The event camera 106 can represent visualinformation in terms of time with respect to a spatial reference in thecamera-pixel arrays. The camera 106 can be a dynamic vision camera 106.Pixels in the camera 106 can respond independently and asynchronously tologarithmic brightness changes in the scene. For a relative motion, astream of events with a microsecond temporal resolution and latency canbe generated and such event representation is in the spatiotemporalform. The optic mirror system 102 can include a camera base 108, a lens109 that can bring light to a fixed focal point, and three mirrors111A-C that can have different orientation placed 45 degrees withrespect to the ray of light coming from the tactile interface 104. Theillumination source 105 (such as an LED ring, LED strip, or other sourceof light) can illuminate the optic mirror system 102 such that the eventcamera 106 is able to observe the brightness changes caused by therelative motion of the object 101. The event camera 106 can beintegrated with the optic mirror system and calibrated to receive thetactile information at the tactile interface 104. The tactile interface104 can be a tool to interact with objects/targets and provide sensoryfeedback about the interaction. The robotic manipulator 100 canfacilitate the mounting of different type of tactile interfaces 104. Weclassify such tactile interfaces 104 as hard and soft fingertips. Thehard fingertip 112 can be a simple transparent rigid layer that offerscheap finger replacements, minimizes wear and tear, and may beunaffected by the weight, material type and geometry of the object 101.The soft fingertip 114 can include deforming skin (opaque skin) orspecialized skin (occluding skin with embedded markers) that canaccommodate a wide range of sensitivities and range of objects 101. Thesoft fingertip 114 and hard fingertip 112 enables the finger to handle awide class of objects 101 such as fragile, soft, light and heavyweightas well as delicate and deformable things like commercial food. Therobotic manipulator 100 can also be customized based on the gripperactuation mechanism, operational environment, targets to be handled,available size of the event camera 106 and requirements of tactilesensing. Both frame-based and event-based cameras 106 can be utilizedfor vision-based tactile sensing. However, various embodiments maypermit availing the neuromorphic sensing capability using the eventcamera.

FIG. 2A shows the robotic manipulator 100 of FIG. 1 in an open state andFIG. 2B shows the robotic manipulator 100 of FIG. 2 in a closed state.The robotic manipulator 100 can have a static part 202 and moving part204 arrangement to reduce the load on the actuation mechanism and toincrease the operational workspace of the robotic manipulator 100. Thestatic part can include an event camera 106, a lens 109, and a mirror111 such that the ray of light passes conveniently without any dynamicmotion of the event camera 106. The rest of the optic mirror system 102can move back and forth to support robotic manipulator 100 states.Moreover, the optic mirror system 102 that works based on laws ofreflection of light upon object contact and the “close” state isillustrated in the left side of the schematic diagram.

The event camera 106 can detect illumination changes that are directlychanneled through optic mirror system from the tactile interface 104.Detection of such transient changes at the tactile interface 104 can beuseful for robotic grasping and manipulation applications. The methodsto measure tactile information with an integrated event camera 106varies, depending on the retrofitted fingertips. The hard fingertip 112can enable the event camera 106 to measure the object contour or patternevents directly when there is a relative object motion. The marker-lesssoft fingertip 114 measures events from skin deformation. Themarker-based soft fingertip 114 provides a closed and controlledenvironment for tactile sensing and measures marker-based events. FIG. 3shows the examples of tactile interfaces 104 for use with the roboticmanipulator 100 of FIG. 1 including the external stimuli at differentfingertips and the perceived tactile information by the event camera106. The illustration shows the accumulated events in a frame, butindividual events have their own timestamp in the spatial space.

FIG. 4 shows an additional example of the robotic manipulator 100, e.g.,viewed in an assembled state from an external perspective in which someinternal components discussed with respect to other views are obscuredfrom view.

FIG. 5 shows a schematic view of the robotic manipulator 100 of FIG. 4 .It is a heterogeneous and modular system, which includes an actuationunit 500, vision-based tactile finger unit 502, and add-on finger unit504. The actuation unit 500 enables the vision-based tactile finger unit502 to move in a parallel motion in relation to the robotic manipulator100 body and facilitates the attachment of custom-made fingers. Thetactile finger unit 502 is integrated with the parallel roboticmanipulator 100 system. Such arrangements provide more operationalworkspace different tactile interface 104 options and reduce graspconstraints for the robot. Moreover, the tactile finger unit 502 candetect transient changes in dynamic scenes (relative object motion) at amicrosecond temporal resolution and acquire non-redundant informationthus enabling efficient processing and real-time response of the roboticmanipulator 100 system. The add-on finger unit 504 can be any soft,rigid or flexible finger or fingertips to support a grasp andapplication needs.

The robotic manipulator 100 can include flexible and compliantmaterials, making them especially suitable for unstructured environmentsand in handling delicate and fragile objects. The compliance andconformity of soft robotic manipulators 100 can allow them to envelopobjects of different sizes and shapes while holding them.

Soft robots can obtain constantly updated information about theirinternal state (proprioception) and external tactile sensing(exteroception) to achieve robust grasping and fine manipulation.Camera-based sensors such as the event camera 106 can provide adequateproprioceptive and exteroceptive information for the robotic manipulator100 and improve its grasping and manipulation abilities.

The event camera 106 can detect transient changes in dynamic scenes interms of brightness intensity. Moreover, event-based cameras have a hightemporal resolution, low latency, efficient data processing and lowpower consumption (e.g., especially when compared to frame-basedcameras). While performing manipulation tasks, timely detection ofproprioceptive and exteroceptive features can enable roboticgrippers/hands to effectively regulate the grip force and maintain astable grasp. Therefore, a robotic manipulator 100 that incorporates asoft compliant finger 502 and a neuromorphic event-based camera sensor106 to refine grasping capabilities and observe proprioceptive andexteroceptive information such that the robot is able to handledifferent types of objects is an attractive option for variousapplications.

In various embodiments, the robotic manipulator 100 integrates aneuromorphic vision-based camera sensor 106 between the two sides of asoft compliant finger to acquire proprioceptive and exteroceptiveinformation. The soft compliant finger 502 can be fabricated from asingle flexible material or a composite of flexible and hard materials.In various embodiments, the side of the finger 502 that interacts withthe objects is fabricated from a flexible material, while the other sidecan either be a flexible or a relatively harder material. The fingerstructure facilitates the mounting of different types of tactileinterfaces, especially, the side that interacts with the object totransfer the information to the camera and a lighting system that isplaced between the finger and the camera to improve the lightingconditions for the detection of brightness intensity. Moreover, the softfinger 502 embodiment could be extended with the optical channel 102similar to the embodiment shown in FIG. 1 . Embodiments may include thesoft finger with tactile capability, integration of the event-basedcamera 106 with a hollow two-sided finger, and the approach foracquiring the necessary proprioceptive and exteroceptive information.The finger can be used as a part of a parallel gripping system thateither incorporates two similar soft fingers or the described softfinger with a conventional rigid finger.

FIGS. 6A and 6B show an example compliant end effector unit 600 that caninclude an end effector 602 with a tactile interface 104 for use with arobotic manipulator 100. The end effector 602 can include a composite ofa flexible and a hard material that can be 3D printed and steel pinsconnecting several joints in a manner that produces soft and passivecompliant envelopment of objects. The steel pins connecting the jointsmay constrain the deformation of the end effector 602, increasing theforce that the end effector 602 can handle. The markers 608A and 608Bare placed on the internal side of the skin and joints, respectively,that interact with the object 101 during the grasping process. Anillumination source 105 can be attached between the end effector 602 andthe camera 106 to provide sufficient lighting for the camera 106. Whiledeforming, the steel pins can preserve the distance between the linkedjoints, allowing the markers to feed information to the camera about theinternal state of the finger, whereas the deformed markers on the skinhelp obtain the curvature of skin deformation and reflect the state ofthe grasped object and its motion, feeding both proprioceptive andexteroceptive information to the camera 106. The event camera 106detects illumination changes in the scene as a stream of events with amicrosecond temporal resolution and latency and processes themcontinuously to produce useful information that can be utilized toachieve robust grasping, regulate the gripper force, and maintain objectstability when uncertainties such as slip occur.

In various embodiments, the soft finger tactile interface 104 is a toolthat can interact with objects/targets and provide sensory feedbackabout the interaction. The robotic manipulator 100 facilitates themounting of different types of soft/flexible skins on the fingerstructure shown in FIG. 6B. The tactile interface 104 can be or includea flexible-hard skin and/or a flexible soft skin. The flexible-hard skincan be or include a transparent rigid flexible layer that offers cheapfinger replacements, minimizes wear and tear, and remains unaffected bythe weight, material-type, and geometry of the object. The soft skinrefers to a deformable opaque skin or a specialized occluded skin withembedded markers. In various embodiments, different soft skins can beutilized to provide customized sensitivity while grasping a wide rangeof objects. The different tactile interfaces and the soft fingerstructure enables a safe, adaptive, and precise handling of a wide classof objects, i.e., fragile, complex, soft, light, heavyweight, etc.,including delicate and deformable items like commercial food. In someembodiments, the soft finger tactile interface can be customized basedon the gripper actuation mechanism, end effector 602 structure,operational environment, targets to be handled, available size of thecamera 106, and requirements of tactile sensing. Both frame-based andevent-based cameras can be utilized for vision-based tactile sensing.However, embodiments may enable availing the neuromorphic sensingcapability for a soft finger without negatively affecting its complianceand conformity.

FIG. 7A shows the end effector 602 of FIG. 6A prior to contacting anobject and FIG. 7B shows the end effector 602 of FIG. 6A aftercontacting the object. It is worth mentioning that conventionalframe-based cameras can be used herein, although neuromorphic sensingcapability may be availed using the event camera.

Joint connections and composite materials are two possible approaches toincrease the feasible range of applied force. The choice of end effector602 design and material selection depends on the application and theforce that will be applied on the end effector 602. For manipulatingextremely soft and small objects, an end effector 602 from homogenousflexible material without joint connections might be sufficient. Suchdesign can be simple and cost-effective. Manipulating a wider range ofobjects of different sizes and materials can involve an end effector 602that is capable of grasping with higher forces. Such capability can beachieved through joint connections and incorporating a hard side of theend effector 602. Moreover, joint connections provide the camera 106with a different kind of information that can help in obtaining theproprioception of the end effector 602. The robotic manipulator 100 canutilize any of these end effector 602 configurations depending on theapplication.

FIG. 8 shows an example robotic manipulator 100 including multiple endeffectors of FIG. 6A, and FIG. 9 shows another example roboticmanipulator 100 including a single end effector of FIG. 6A. Theactuation unit enables the end effector 602 to move in parallel motionin relation to the gripper body and facilitates the attachment ofcustom-made end effectors 602. The compliant end effector unit 600 caninclude the camera 106, illumination source 105 and soft end effector602 as shown in FIG. 6A. With the end effector 602 structure, theparallel motion induced by the actuation unit is enough to allow the endeffector 602 to envelop the object and adapt to its shape. The add-onend effector unit 504 can be any soft or rigid end effector 602. FIG. 8shows a homogenous approach where the add-on end effector unit 504includes a soft finger 602B similar to soft finger unit 602A, whereasFIG. 9 shows a heterogenous approach with an add-on end effector unit504 that has a hard end effector 900 that can be equipped withconventional sensors. In various embodiments, the robotic manipulator100 is able to handle objects that may be delicate, fragile, deformableand/or lightweight. The design and/or sensing ability of the roboticmanipulator 100 may overcome limitations of precision, robustness,scalability, and controllability, for example. In some embodiments, therobotic manipulator 100 may exhibit technological advancements towardsbetter perception capabilities, slip detection and suppression, objectpose estimation, force estimation and control. The robotic manipulator100 may be suitable for factory automation of food, toy, meat andgrocery industry. The robotic manipulator 100 may be suitable to handleobjects especially in service sector and human-centered environments.The robotic manipulator 100 may utilize a passive soft finger withneuromorphic vision-based sensing and skin and joint markers tosuperiorly extract proprioceptive and exteroceptive information.

In this disclosure, various embodiments can include a sensorconfiguration that can provide pose and contact force estimations usinga single camera 106. As shown in FIG. 10 , light is propagated to thecamera 106 from two sources. The first source passes light from theexternal environment into the camera 106 and thus enables visualfeedback on surroundings of the machining tool 1000. This visualfeedback is useful for the initial positioning of the tool using avariety of visual guidance and servoing techniques. The second source oflight is internal and is used to provide tactile information during themachining process. Light is first reflected from the inner surface of asoft-tactile interface 104 prior to propagating to the camera 106through a set of mirrors. Several visual markers 608 are distributedalong the inner surface of the tactile interface 104 in a known pattern.When the machining tool 1000 achieves contact with an external body, thetactile interface 104 deforms and the markers' distribution patterns arealtered. By observing the deformation of the tactile interface 104through changes in the visual markers' pattern, contact forces and posecan be estimated.

FIG. 10 shows an example in which the tactile interface 104 includes anelastic spherical part. The spherical part may be installed in closeproximity with an inner camera observing the elastic material's innersurface. The inner surface may be made from silicon or other suitablyelastic material. The spherical part may be fabricated through moldingor other suitable construction technique. The spherical part may includeseveral pockets distributed along its inner surface. Black sphericalbeads can be inserted in those pockets and a thin layer of transparentsilicone brushed on the inner side of the sensor to create a thin layerthat will hold the beads in place, for example.

FIG. 10 shows an example robotic manipulator 100 that can include anoptical sensor and a tactile sensor. The robotic manipulator 100 caninclude an optic mirror system 102 as described in FIG. 1 . The opticmirror system 102 can include a set of mirrors 1004A-D. Some mirrors1004A, 1004B can be used to direct light for observing externalsurroundings, while other mirrors 1004C, 1004D can be used to directlight for observing tactile information. Light can be directed throughone or more lenses 109A, 109B to reach the camera 106. Using the set ofmirrors 1004A-D, the same camera 106 can simultaneously monitor externalsurroundings and provide tactile information by observing thedeformation in a soft tactile interface. Such a sensor configuration canoffer considerable advantages over conventional perception methodscurrently used for machining applications in terms of cost, powerconsumption, maintainability, and flexibility. As only a single sensingelement is needed, production cost and power consumption aresignificantly reduced and no sensor synchronization procedure isrequired. Additionally, most parts of the sensor can be easilyfabricated or 3D printed with a low cost. As such, the presentedinterface can be easy to maintain and offer great customizationpotential to fit specific requirements by simply adjusting the sensor'sdimensions, placement, and tactile interface. For instance, as thetactile interface 104 can be separated from the sensing element,different tactile interfaces with different stress-strain properties canbe easily replaced offering a low-cost trade-off between resolution andrange for the contact force estimation problem. In a similar manner, thesensor can be customized in terms of the number of contact points withtactile feedback. FIG. 11 shows an example end effector with tactilesensors. As illustrated in FIG. 11 , the visual servoing element can beswapped for another tactile interface; which passively stabilizes themachining tool 1000 and enables the sensor to observe contact torques.The same concept can be further extended to three and more contactpoints as showcased in FIG. 12 , which shows another example endeffector with tactile sensors 104. Another customization possibility isthe number of cameras utilized in the sensor. As manufacturers areracing to develop smaller and more efficient cameras, the sensor can beadjusted to fit multiple cameras each for a specific sensing task.

Various embodiments herein can be customized for multi-point tactilefeedback. For example, as shown in FIG. 11 , a two-point tactile sensingconfiguration using a single camera is depicted. The same concept can beextended for more than two contact points in 3D as shown in FIG. 12 .The sensor design can embed multiple cameras 106A, 106B for easierfabrication of the sensor housing, for example, as shown in FIG. 13 .The multiple cameras 106A and 106B can receive light directed by lenses109A and 109B respectively. Each of the lenses 109A-B can focus lightinto their respective camera 106A-B. This design can be feasible asmanufacturers develop smaller and more efficient cameras. Embodimentsmay hold value for a wide spectrum of machining processes such asdrilling, deburring, and grinding up to electronics manufacturing. Whilethe specific machining tool 1000 can be different, the fundamentalconcepts of the sensor may remain the same. To better illustrate thepresented concept, FIG. 14 presents a typical use case of the sensor ina robotic drilling scenario.

FIGS. 14A through 14C illustrate an example use case of the sensor in aconfiguration for robotic drilling applications. FIG. 14A shows thesensor is equipped in a machining tool 1000 with a drill bit. FIG. 14Bshows the drill bit driven towards initial alignment with the workpieceusing visual feedback from the aperture of the camera 106. FIG. 14Cshows once the drill bit of the machining tool 1000 is aligned, it isdriven towards the target workpiece which causes the tactile interface104 to deform. The camera 106 observes this deformation through thedisplacement of several pre-defined markers within the tactile interface104. Contact forces can then be estimated from the observed deformationusing computer vision and machine learning algorithms. For example, thecontact forces can be estimated using a fully-connected network, aconvolutional neural network, a long short-term memory convolutionalneural network, or any other suitable deep learning method. In additionto a contact force, the deep learning method can also be used toestimate a contact angle or a vibration associated with the deformation.In some embodiments, the sensor may provide information for normalityenhancement, e.g., such that the contact angle can be measured and thetool re-aligned to be normal relative to the workpiece in response. Thesensor can be generic in terms of the type of the utilized camera 106.However, dynamic vision cameras (Neuromorphic) which output the changesin brightness independently and asynchronously for each pixel would be abetter fit for the required optical tasks due to their microsecond levellatency. The high dynamic range of dynamic vision cameras can also be abeneficial feature for the sensor as light is propagated to the camerafrom multiple sources that might have different lighting conditions andcontain visual features at a widely different depth level. The highdynamic range of the neuromorphic camera 106 enables perception withvery small apertures, which promote an exceptionally wide depth offield. Neuromorphic cameras 106 may be less susceptible to under/overexposure and motion blur and only require milliwatt level power. Majormanufacturers are also investing heavily in dynamic vision technologiesto produce smaller, cheaper, and higher resolution cameras; which canmake the use of dynamic vision sensors in industrial application moreeconomically and technically feasible.

FIG. 15 is a diagram that illustrates an example of a neural network1500 that can be used for estimating data associated with the tactileinterface 104. The neural network can include an input layer 1501 thatcan receive inputs associated with the tactile interface 104. In someexamples, the input layer 1501 can receive one or more outputs from asoftware simulation 1502 of the tactile interface 104. In some examples,the software simulation 1502 can be a finite element analysis softwaresimulation 1502. The finite element analysis may correspond to analysisof a digital twin of the tactile interface 104, for example. Thesoftware simulation 1502 can generate synthetic training data that canbe used to train the neural network 1500. The input layer 1501 of theneural network 1500 can additionally or alternatively receive an image1503 from the camera 106. The neural network 1500 can include one ormore hidden layers 1504 that can process data from the input layer 1501.The one or more hidden layers 1504 can transmit data to an output layer1506, where it can be used to estimate data associated with the tactileinterface. For example, outputs generated by the output layer caninclude estimates of a contact force associated with the tactileinterface 104, a contact angle associated with the tactile interface104, a vibration measurement associated with the tactile interface 104,or any combination thereof.

Based on the disclosure and teachings provided herein, a person ofordinary skill in the art will appreciate other ways and/or methods toimplement the various embodiments. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. It will, however, be evident that various modifications andchanges may be made thereunto without departing from the broader spiritand scope of the disclosure as set forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit thedisclosure to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructions,and equivalents falling within the spirit and scope of the disclosure,as defined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments of the disclosure anddoes not pose a limitation on the scope of the disclosure unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe disclosure.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is intended to be understoodwithin the context as used in general to present that an item, term,etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y,and/or Z). Thus, such disjunctive language is not generally intended to,and should not, imply that certain embodiments require at least one ofX, at least one of Y, or at least one of Z to each be present.

Various embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the disclosure.Variations of those embodiments may become apparent to those of ordinaryskill in the art upon reading the foregoing description. The inventorsexpect skilled artisans to employ such variations as appropriate and theinventors intend for the disclosure to be practiced otherwise than asspecifically described herein. Accordingly, this disclosure includes allmodifications and equivalents of the subject matter recited in theclaims appended hereto as permitted by applicable law. Moreover, anycombination of the above-described elements in all possible variationsthereof is encompassed by the disclosure unless otherwise indicatedherein or otherwise clearly contradicted by context.

What is claimed is:
 1. A robotic manipulator, comprising: a first endeffector comprising a contact surface configured to make physicalcontact with an object; and an event camera that is couplable with therobotic manipulator and configured to receive optical data associatedwith a visually-detectable deformation of visual markers distributed onthe contact surface in response to making contact with the object. 2.The robotic manipulator of claim 1, wherein the contact surfacecomprises a flexible or semi-flexible material and is further configuredto deform in response to contacting the object.
 3. The roboticmanipulator of claim 1, wherein the first end effector further comprisesa mirror positioned to direct the optical data associated with theobject from the contact surface to the event camera.
 4. The roboticmanipulator of claim 3, wherein the contact surface comprises visualmarkers distributed on an exterior of the contact surface and theoptical data comprises deformation of the visual markers in contact withthe object.
 5. The robotic manipulator of claim 4, wherein the first endeffector further comprises a light source positioned to direct lightthrough the contact surface and illuminate the visual markers.
 6. Therobotic manipulator of claim 1, further comprising: a machining toolcoupled with the robotic manipulator; and a second end effector.
 7. Therobotic manipulator of claim 6, wherein the second end effectorcomprises a second end effector configured to engage with the object. 8.The robotic manipulator of claim 6, wherein the second end effectorcomprises a lens configured to focus light from an external environmentand a mirror configured to direct the light from the externalenvironment to the camera.
 9. The robotic manipulator of claim 6,wherein the camera is a first camera and is aligned with the first endeffector and the robotic manipulator further comprises a second cameraaligned with the second end effector.
 10. The robotic manipulator ofclaim 6, further comprising a third end effector, wherein the second andthird end effectors comprise respective second and third contactsurfaces configured to engage with the object.
 11. A roboticmanipulator, comprising: a machining tool; a first end effectorcomprising a contact surface configured to engage with the object; acamera coupled with the robotic manipulator and configured to receiveoptical data associated with a visually-detectable deformation of visualmarkers distributed on the contact surface in response to making contactwith the object; and a second end effector positioned on an opposingside of the machining tool from the first end effector and comprising alens configured to focus external light and a mirror positioned todirect the focused external light to the camera.
 12. The roboticmanipulator of claim 11, wherein the contact surface comprises tactilemarkers positioned on an exterior of the contact surface in a pattern,the tactile markers configured to deform in response to contact with theobject.
 13. The robotic manipulator of claim 12, wherein the first endeffector comprises a light source positioned to emit light through thecontact surface and illuminate the tactile markers.
 14. The roboticmanipulator of claim 11, wherein the camera comprises an event camerathat comprises a plurality of photodiodes that are configured toasynchronously output data values corresponding to light intensityvalues of the photodiodes.
 15. A method, comprising: engaging a contactsurface of a first end effector of a robotic manipulator to an object;and receiving, from an event camera that is coupled with the roboticmanipulator, optical data associated with a visually-detectabledeformation of visual markers distributed on a contact surface inresponse to making contact with the object.
 16. The method of claim 15,further comprising: deforming the a flexible or semi-flexible materialof the contact surface in response to contacting the object.
 17. Themethod of claim 15, further comprising directing the optical dataassociated with the object from the contact surface to the camera. 18.The method of claim 17, further comprising determining an amount ofdeformation of the contact surface based on a variation of visualmarkers on the contact surface as represented in the optical data. 19.The method of claim 18, further comprising: positioning a light sourceto direct light through the contact surface; and illuminating the visualmarkers.
 20. The method of claim 19, further comprising: coupling amachining tool with the robotic manipulator; and coupling a second endeffector with the robotic manipulator.